Make Helix Job Monitor uploads restart-safe - #17179
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens the Helix Job Monitor’s Azure DevOps test-result upload pipeline so it can be safely restarted after crashes/timeouts by separating durable completion (tagged runs) from queued/in-progress/failed uploads, and by bounding/targeting retries to avoid replaying ambiguous Azure DevOps writes.
Changes:
- Introduces explicit upload lifecycle state tracking (queued/in-progress/durably-completed/failed) and updates the runner/queue to honor durable completion tags.
- Adds transient-failure classification and bounded retries for safe (read-only) download operations while disabling retries for state-changing Azure DevOps operations.
- Expands unit coverage across runner, services, publisher, and fakes for restart-safety and transient/permanent failure behaviors.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/JobMonitorRunnerTests.cs | Adds/updates scenarios asserting restart-safe behavior and non-replay of ambiguous uploads/completions. |
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/HelixServiceTests.cs | Adds coverage for transient per-file download failures while still attempting the full batch. |
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/Fakes/FakeHelixService.cs | Adds queued transient download failure injection to simulate retries. |
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/Fakes/FakeAzureDevOpsService.cs | Adds create/complete failure injection and tracks complete call counts for new behaviors. |
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/AzureDevOpsServiceTests.cs | Verifies create/complete do not retry ambiguous writes. |
| src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/AzureDevOpsResultPublisherTests.cs | Adds tests for configurable retry behavior for ambiguous writes. |
| src/Microsoft.DotNet.Helix/JobMonitor/TransientFailureDetector.cs | New shared transient-failure classifier used by queue/services. |
| src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs | Splits upload into phases with bounded retries for safe operations and non-replay of ambiguous writes. |
| src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs | Treats transient per-file failures as retryable after attempting remaining files; throws aggregated transient failure. |
| src/Microsoft.DotNet.Helix/JobMonitor/Services/AzureDevOpsService.cs | Disables adapter-level transient retries for create/complete/attachment writes; disables publisher write retries; improves HttpRequestException status propagation. |
| src/Microsoft.DotNet.Helix/JobMonitor/MonitorState.cs | Replaces “processed” set with explicit upload lifecycle state machine keyed by Helix job. |
| src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.Design.md | Updates behavioral spec to document restart-safe semantics and retry/non-replay rules. |
| src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.cs | Avoids marking jobs as processed at queue time; prevents duplicate enqueue within an invocation via state. |
| src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/Model/AzureDevOpsReportingParameters.cs | Adds RetryWrites parameter to control write retry behavior. |
| src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/AzureDevOpsResultPublisher.cs | Implements configurable retry count based on request method / write-retry configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/AzureDevOpsResultPublisher.cs:553
GetRetryCountchecksmethod == HttpMethod.Get, which can miss logically-GET methods created vianew HttpMethod("GET")(or other equivalent instances). This can incorrectly disable retries for GET requests whenRetryWritesis false; compare by value instead of reference.
internal static int GetRetryCount(HttpMethod method, bool retryWrites)
=> retryWrites || method == HttpMethod.Get ? 10 : 0;
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/AzureDevOpsResultPublisher.cs:553
- GetRetryCount uses
method == HttpMethod.Get, which is reference equality forHttpMethod. This returns false fornew HttpMethod("GET")(and any non-cached instance), disabling retries for safe read-only GETs and causing the new unit test to fail. UseEquals(or comparemethod.Method) instead.
internal static int GetRetryCount(HttpMethod method, bool retryWrites)
=> retryWrites || method == HttpMethod.Get ? 10 : 0;
Track upload lifecycle explicitly, retry only safe read operations, avoid replaying ambiguous writes, and keep incomplete runs untagged for later recovery.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
11e4992 to
6c5f6ef
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs:27
- The file header summary still says uploads have “indefinite retry on transient errors”, but the new implementation retries only bounded read-only operations and attempts state-changing AzDO operations once (to avoid replaying ambiguous writes). Updating the summary here will prevent confusion later.
/// </summary>
internal sealed class TestResultUploadQueue
{
private const int MaximumTransientAttempts = 3;
private const string AzdoWarningPrefix = "##vso[task.logissue type=warning]";
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs:225
- The warning log message says “The run remains untagged … may retry the upload” even when the failure happens before any test run is created (testRunId=0), which is confusing. Consider emitting a different message when no run exists yet.
_logger.LogWarning(ex,
"{Prefix}Failed to {OperationDescription} for job {JobName}. Test run ID was {TestRunId}. "
+ "{FailureKind} The run remains untagged and a later monitor invocation may retry the upload.",
AzdoWarningPrefix,
operationDescription,
Pass retry counts explicitly, use the shared retry helper with transient classification, and preserve reporting parameter compatibility while disabling ambiguous write retries.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
Describe bounded read retries and single-attempt writes accurately.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
|
Addressed the retry feedback in
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs:235
- The warning message assumes "The run remains untagged" even when no test run has been created yet (e.g., download failures with testRunId==0), and it also asserts an untagged state that may not be true after an ambiguous completion/tagging failure. This can mislead troubleshooting; consider wording that’s conditional/hedged while still indicating that a later monitor invocation may retry the upload.
? "Transient retry limit reached."
: "The operation may have partially completed and is not safe to replay in this invocation."
: "The failure is not retryable.";
_logger.LogWarning(ex,
"{Prefix}Failed to {OperationDescription} for job {JobName}. Test run ID was {TestRunId}. "
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs:235
- The warning message here always mentions a "test run" and "The run remains untagged" even when
testRunIdis 0 (e.g., during the download phase, before any run is created). That makes the log misleading and can confuse diagnosing failures.
Consider tailoring the message when no run exists yet (or omitting the test-run clause entirely for non-AzDO operations).
_logger.LogWarning(ex,
"{Prefix}Failed to {OperationDescription} for job {JobName}. Test run ID was {TestRunId}. "
+ "{FailureKind} The run remains untagged and a later monitor invocation may retry the upload.",
AzdoWarningPrefix,
operationDescription,
src/Microsoft.DotNet.Helix/JobMonitor/MonitorState.cs:170
AddProcessedHelixJobsseeds durable completion state from existing AzDO tags, but it does not increment_processedJobCount. As a result,StatusReporterwill treat these jobs as processed (viaIsHelixJobProcessed), while the final summary usesProcessedJobCountand may report 0 processed jobs even when everything was already processed before this invocation.
public void AddProcessedHelixJobs(IEnumerable<string> jobNames)
{
lock (_sync)
{
foreach (string jobName in jobNames)
{
_testResultUploadStates[jobName] = TestResultUploadState.DurablyCompleted;
}
Make RetryWrites a normal optional record constructor parameter without compatibility scaffolding for this newly introduced option.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs:235
- The warning emitted on failure always says "The run remains untagged" even when the failure occurs before any test run exists (e.g., download failures where testRunId is 0). This can mislead diagnosis in AzDO logs. Consider wording this in terms of "durable tagged completion" rather than implying an existing run.
_logger.LogWarning(ex,
"{Prefix}Failed to {OperationDescription} for job {JobName}. Test run ID was {TestRunId}. "
+ "{FailureKind} The run remains untagged and a later monitor invocation may retry the upload.",
AzdoWarningPrefix,
Apply explicit retry counts and transient classification through Microsoft.Arcade.Common.ExponentialRetry instead of extending the publisher retry helper.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
Delete the unused metadata pipeline and its private helpers rather than maintaining commented, nonfunctional scaffolding.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: baa969a4-1d79-4b6a-a3f0-4d349d768b5e
|
Updated in \70f6a1c37\ and \31dff43a7: upload read retries now use \Microsoft.Arcade.Common.ExponentialRetry, with the callback deciding whether a transient failure is retryable and explicit retry-count semantics retained. The custom \RetryHelper\ extension was removed. I also deleted the inactive test-metadata method, commented block, and private helpers rather than maintaining nonfunctional scaffolding. |
Summary
Stacking
This PR is stacked on #17178. Until that PR merges, its two documentation commits appear in this PR as well; the implementation commit is
Make Job Monitor uploads restart-safe.Closes #17173
Parent epic: #17171